Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning

1 INTRODUCTION

• MOPD assigns teachers by prompt-level domain labels and weights all response tokens uniformly, so it cannot separate useful expert knowledge from erroneous reasoning within one answer.The section frames the unresolved question as deciding both which expert may teach and where its supervision should apply.Source: §1 · pp. 1–2
• VG-OPD licenses an expert only when it has a criterion-specific advantage over the student, then uses their disagreement to select tokens for supervision.The shared criterion verifier evaluates expert and student answers against the same requirement.Source: §1 · p. 2
• Across seven benchmarks, VG-OPD improves 4B accuracy over rubric-GRPO by 2.0 points on scientific reasoning and 3.3 on domain science; 8B shows the same trend and strongest overall performance.Ablations confirm that verifier-based expert eligibility and selective token supervision are critical to the gains.Source: §1 · p. 2
• The paper introduces criterion-level eligibility and reports controlled evidence that both expert eligibility and token-level placement contribute to improvements across heterogeneous scientific reasoning benchmarks and model scales.The evaluation uses four RL-trained capability experts and 4B and 8B students across seven benchmarks.Source: §1 · p. 2

2 PRELIMINARIES

• A rubric represents each prompt with verifiable criteria, their verifiers, and signed weights; GRPO optimizes a scalar reward that sums the criteria.The scalar reward loses criterion-level structure, although the rubric also identifies failed criteria and allows the same verifier to score another answer.Source: §2 · pp. 2–3
• OPD matches the student to a teacher on the student’s own rollouts, while MOPD assigns one domain teacher per prompt and weights every token equally.MOPD is a weighted token-level OPD case with fixed routing before the rollout is observed.Source: §2 · p. 3
• VG-OPD replaces fixed teacher assignment and token weights with verification-guided routing and sparse weighting for each failed criterion in a rollout.For each such criterion, it decides which expert may teach, which tokens receive supervision, and how strongly.Source: §2 · p. 3

3.1 OVERVIEW: WHO, WHERE, AND HOW MUCH

• VG-OPD adds a weighted, token-level teacher advantage to the student’s task-reward advantage during the policy update.For each rollout and failed criterion, the axis-matched expert teaches only if its answer scores higher; supervision is limited to tokens where it disagrees with the student.Source: §3.1 · p. 3

3.2 WHO TEACHES: LICENSING AND ROUTING

• An expert is licensed for a failed criterion only when its cached capability probe beats the student under that criterion’s verifier; pitfall criteria, abstentions, and sufficiently high-reward rollouts receive no teacher term.Licensing and routing are per criterion, so a rollout can receive teachers from multiple axes.Source: §3.2 · pp. 3–4

3.3 WHERE TO TEACH: DISAGREEMENT LOCALIZATION

• Each teacher’s distillation mask marks tokens where it would not have written the student’s response, dilated by one token on each side.The rank-based mask marks about η of a response even when the teacher largely agrees; the gate, not the mask, establishes that the teacher has something to teach.Source: §3.3 · p. 5

3.4 HOW MUCH: VERIFIED-GAIN WEIGHTS

• Each token receives supervision from at most one teacher, selected by the largest verified weight among licensed criterion masks; tokens outside all masks receive none.Overlapping masks therefore do not produce conflicting gradients.Source: §3.4 · p. 5
• GRPO combines task-level progress with gated distillation because the distillation advantage is nonpositive everywhere; the task-reward stream counterbalances it while the sparse gate limits its reach.Applying the distillation advantage alone can continually suppress probability mass and accelerate entropy collapse.Source: §3.4 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation uses 4B and 8B students, six baselines, and seven benchmarks grouped into scientific reasoning, domain science, and general reasoning.The baselines include rubric-GRPO, OPSD, MOPD, teacher-anchored GRPO, and CriPO.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• VG-OPD ranks first on five of seven benchmarks at both scales and is within half a point on MMLU-Pro and MATH500.Its gains concentrate on knowledge-intensive science while general reasoning is preserved.Source: §4.2 · p. 6

4.3 ABLATION STUDIES

• Dense distillation lowers GRPO below its 53.0 scientific-reasoning floor, while gated distillation adds +2.0 / +3.3 / +0.2 across the three benchmark groups.Component ablations show the largest loss comes from a coverage-matched random mask: 6.3 / 12.1 / 5.6 points across scientific, domain, and general reasoning.Source: §4.3 · p. 7, Table 3 · p. 7, Table 2 · p. 7

4.4 FRAMEWORK ANALYSIS

• Teacher-disagreement positions show a larger decrease in the starting model’s token log-probability than highest-surprisal or random positions, while 88% of licensed criteria go to the prompt’s own-axis expert.Figure 3 also distinguishes the two mechanisms: token-position comparisons and expert-routing counts.Source: §4.4 · pp. 7–8, Figure 3 · p. 8

4.5 CASE STUDY

• On a held-out RaR-Science question, the GRPO+VG-OPD student passes every criterion, whereas the GRPO+MOPD student still fails the answer.VG-OPD licenses experts criterion by criterion and has each licensed expert supervise only tokens where it disagrees with the student.Source: §4.5 · p. 8

• Multi-teacher OPD merges RL-trained specialists, with prior approaches routing by domain, teacher confidence, example reliability, or teacher conflicts.MOPD routes each prompt by its domain label and applies uniform token-level reverse KL.Source: §5 · p. 9
• Selective methods gate or choose supervision at rollout, prompt, token, or region level, but the surveyed approaches do not check teacher success on each criterion.The comparison spans verifier gates, token selectors, and supervision routed between teacher and student.Source: §5 · p. 9
• Rubrics support scientific RL, criterion-to-verifier routing, rubric synthesis, and several forms of self-distillation or rubric-scored OPD.CriPO decomposes rubric-RL by criterion and localizes token-level self-distillation.Source: §5 · p. 9

• VG-OPD uses an expert’s counterfactual gain on a criterion to license, route, and weight its signal, then uses disagreement with the student to localize it.The method is incorporated into a standard GRPO update.Source: §6 · p. 9
• On scientific reasoning, VG-OPD outperforms uniform multi-teacher distillation, privileged self-distillation, dense fusion, and rubric-based RL.The conclusion also reports completion where dense distillation does not.Source: §6 · p. 9
• The conclusion attributes VG-OPD’s gains to where it teaches rather than how much, and expects this question to matter when merging specialists elsewhere.Source: §6 · p. 9

• The seven benchmarks are text-only and concentrate on physics, chemistry, biology, medicine, and mathematics.Other scientific domains and figure, molecular-structure, or experimental-data modalities are outside the study and would require extending the criteria bank and verifiers.Source: LIMITATIONS · p. 9
• The study evaluates 4B and 8B students; whether stronger students benefit similarly remains to be verified systematically.Source: LIMITATIONS · p. 9

A.1 SIGN AND SCALE OF THE TEACHER STREAM

• The teacher stream is non-positive and only suppresses probability mass, so the task-reward stream remains in GRPO while gating and masking support stability.For a fixed weight before clamping, its expected value is −w_t KL_t; the rollout pressure is bounded by the mask’s maximum gate strength.Source: §A.1 · p. 12

A.2 IMPLEMENTATION DETAILS

• A rollout hook runs after reward computation and before teacher scoring, attaching token weights and each token’s teacher log-probability to the batch.It runs probes, gating, routing, and localization asynchronously per sample; internal errors zero-fill weights rather than stall training.Source: §A.2 · p. 13

B.1 TRAINING DATA

• Training data combines four Dr.SCI-derived datasets, and the mixture is non-commercially encumbered, so it is used for research only without relicensing.Closed-form items use derive-then-verify; open-ended items use the rubric-import route.Source: §B.1 · p. 13

B.2 CRITERIA BANK

• Criteria come from verified intermediate steps for closed-form items or imported rubrics for open-ended items, with unusable criteria filtered and penalty criteria retained at roughly 16%.Items lacking a positive-weight criterion on their primary axis are dropped.Source: §B.2 · p. 13

B.3 VERIFIERS

• The verifier registry combines deterministic checks and an LLM judge, whose calibration found a 39% false-trigger rate on clean penalty-item answers.Because of this weakness, pitfall criteria are excluded from gating and receive small absolute weights in rewards.Source: §B.3 · p. 14

B.4 CAPABILITY EXPERTS

• Each capability expert is a rank-64 LoRA adapter trained with GRPO on its axis-filtered data and its own axis reward.Quantitative and symbolic experts use deterministic verifier rewards; mechanistic and evidence experts use graded judge rewards.Source: §B.4 · p. 14

B.5 TRAINING DETAILS

• VG-OPD runs use one configuration for both 4B and 8B students, while baselines share it and differ only in teacher construction.Ablation arms change exactly one switch each.Source: §B.5 · p. 14
• The thinking-mode start saturated at 0.585–0.588 on the strongest axis, while the non-thinking start rose from 0.541 to 0.666 in 200 steps and was used in the main text.With the thinking-mode start, no expert outscored the student under criterion verifiers, so the gate licensed almost nothing.Source: §B.5 · p. 14
• All reported checkpoints are offline-scored winners among the top-k in-training candidates, selected on the internal dev set before external evaluation.In-training validation reordered the final candidate ranking in most arms.Source: §B.5 · p. 14

B.6 COMPUTE COST

• VG-OPD runs at a median 146.8 s/step versus 94.7 s for the RL-only floor, with the added time concentrated in generation rather than the update phase.The update phase is 26.0 versus 19.8 s, and MOPD and CriPO run at 70–104 s/step without a probe stage.Source: §B.6 · p. 15

B.7 BASELINES

• Distillation baselines share VG-OPD’s student, data, budgets, and optimizer settings, while the RL-only floor disables distillation and uses a stable-optimal no-entropy configuration.The floor uses graded rubric rewards; the authors note that pure GRPO’s entropy diverges under the entropy bonus tolerated by distillation arms.Source: §B.7 · p. 15
• MOPD uses prompt-level routing, uniform token weights, and no verification with the same four-expert pool; OPSD uses the rubric-conditioned base as teacher with uniform weights.These baseline constructions distinguish merging recipes from teacher-pool quality or privileged rubric context.Source: §B.7 · pp. 15–16
• CriPO uses suppressed criteria and a counterfactual self-teacher, while teacher-anchored GRPO uses dense reverse-KL regularization from a rubric-conditioned base teacher.CriPO’s counterfactual context doubles sequence length, so that arm trains with an enlarged context window.Source: §B.7 · p. 16

B.8 BENCHMARK DETAILS

• Evaluation uses greedy non-thinking decoding with at most 4,096 new tokens across seven public benchmarks in scientific, domain-science, and general-reasoning groups.The harness scores answers with task-matched methods and serial judging; 74 overlapping RaR-Science test items were removed, and other benchmarks were checked by exact and prefix matching.Source: §B.8 · p. 16

C.1 TRAINING-PARADIGM AND ABLATION COMPARISONS

• Only GRPO+VG-OPD exceeds the RL floor on all three benchmark groups; among standalone recipes, only VG-OPD reaches the floor on both science groups.The dense recipes OPSD and MOPD remain at or below the floor in every group when used inside GRPO.Source: Figure 5 · p. 17, §C.1 · p. 17
• The coverage-matched random mask causes the largest ablation drop on every group, including −12.1 points on domain science, while flat weights cost least but still lose 3.6 points on scientific reasoning.The figure labels bars with absolute accuracy from Table 3.Source: Figure 5 · p. 17, §C.1 · p. 17
• Gate and routing variants cost most on scientific reasoning, whereas indiscriminate dense recipes remain at or below the RL floor across groups.Figure 5 presents the group averages as differences from the RL floor and full VG-OPD, respectively.Source: Figure 5 · p. 17, §C.1 · p. 17

C.2 CASE STUDIES

• The case studies use held-out questions selected for clear teacher disagreement; the physics example is one of four starting-checkpoint samples, two of which already pass every criterion.Texts are cached greedy outputs, and verdicts come from the training verifier.Source: §C.2 · p. 17
• On the medicine rollout, overlapping disagreement masks are assigned by verified weight, leaving 97 tokens to the evidence expert and 40 to the mechanistic expert.The masks overlap on 60 tokens; the second teacher’s exclusive share is modest.Source: §C.2 · p. 18, Figure 7 · p. 20
• VG-OPD’s student passes all six physics criteria, while MOPD’s student fails c1 and c5 after distilling the same expert across all tokens.The expert is licensed only on c2 and c6, where it beats the rollout, and refused on c1 and c5.Source: §C.2 · pp. 17–18, Figure 6 · p. 19