OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning

Motivation

• Existing approaches face costly dynamic search-trajectory construction and expensive, unstable task-specific teacher training.Direct OPD with an off-the-shelf teacher also suffers from teacher–student distributional misalignment.Source: §1 · p. 1

Method

• OPDSearch+ uses a frozen off-the-shelf teacher to distill student-generated live-search trajectories without task-specific teacher training.The teacher supplies token-level supervision for reasoning decomposition and evidence integration.Source: §1 · pp. 1–2
• The two-stage pipeline applies RL after distillation, refining the student from a richer behavioral foundation than RL from scratch.The paper frames distillation as reshaping the student policy for subsequent RL.Source: §1 · p. 2

Results

• 13.1% and 8.5% gains are reported on HotpotQA and 2WikiMultihopQA, respectively, over the best 3B baseline AutoRefine-Base.These are the paper’s highlighted multi-hop improvements.Source: §1 · p. 2
• 0.4402 mean EM across seven QA benchmarks exceeds all 3B RL baselines, including GiGPO-Instruct at 0.421.The result is reported for the 3B OPDSearch+ model.Source: §1 · p. 2

RL for Search-Augmented Reasoning

• Search-R1 and subsequent methods train search agents with outcome-only rewards, while process-reward methods add step-level supervision with additional engineering.Examples include GPT-generated sub-questions and contrastive group construction.Source: §2 · p. 2

Knowledge Distillation for Reasoning LLMs

• Offline distillation suffers from train–test distribution mismatch because students train on teacher trajectories but infer from their own distributions.On-policy distillation addresses this by scoring student-generated samples with the teacher.Source: §2 · pp. 2–3
• Recent OPD work studies how KL direction and entropy adaptation affect training stability under on-policy distribution mismatch.The cited methods include reverse-KL, entropy-adaptive mixing, and analyses identifying forward KL as important for stability.Source: §2 · p. 3

KD-RL Hybrid Methods

• KD–RL hybrids combine distillation with reinforcement learning, but the cited methods operate in static text-generation settings rather than interactive retrieval.Examples include advantage-weighted distillation, proximal OPD, teacher-context generation, and self-conditioned KL.Source: §2 · p. 3

Implicit Supervision

• Teacher-favored tokens receive amplified gradients up to Rmax, providing implicit query-quality supervision without a dedicated retrieval reward.This per-token weighting can distinguish teacher-preferred entity-specific queries from low-value question copying.Source: §3.1 · p. 3, §3.3 · p. 4

Distribution Shift

• On-policy distillation reduces retrieval-context mismatch because prefixes are induced by the student interacting with the retriever rather than pre-collected teacher trajectories.Search results depend on prior queries, so policy changes can cascade into different retrieved passages and later reasoning contexts.Source: §3.1 · pp. 3–4, §3.2 · p. 4

On-Policy Search Distillation

• OPDSearch+ distills a frozen teacher on student-generated trajectories that interleave reasoning, queries, retrieved passages, and answers.The student interacts with a live search engine, while retrieved passages are environment-provided rather than model-generated.Source: §3.3 · p. 4, §3.2 · p. 4

RL Refinement

• The objective combines forward-KL distillation with KL regularization against the frozen initial student checkpoint, while teacher-derived token rewards motivate subsequent RL refinement.The teacher signal is dense and stable but reflects teacher preferences rather than task-specific outcomes.Source: §3.3 · p. 4, §3.4 · p. 4

Forward-KL Objective

• Forward KL is estimated with teacher-to-student importance ratios on model-generated token positions, excluding retrieved-passage tokens through state masking.The ratio is clipped between ϵ and Rmax, limiting importance-weight magnitude while introducing controlled bias relative to the exact gradient.Source: §3.3 · p. 4

Objective Choice

• Reverse-KL surrogates can collapse because their unbounded high-coefficient terms are frequently sampled when the student over-assigns probability relative to the teacher.Forward-KL instead caps high ratios on tokens that are rarely sampled under the student policy, reducing the frequency of high-magnitude contributions.Source: §3.1 · p. 3, §3.3 · p. 4

Main results

• OPDSearch+ with a 3B model achieves mean EM 0.4402, surpassing GiGPO-Instruct at 0.421 and all other 3B methods.Source: §4.2 · p. 5

Single-hop QA

• OPDSearch+ exceeds AutoRefine-Base on single-hop QA with SH-Avg 0.520 versus 0.512.Its individual scores are NQ 0.4852, TriviaQA 0.6226, and PopQA 0.4513.Source: §4.2 · p. 5

Multi-hop QA

• +13.1% on HotpotQA and +8.5% on 2WikiMultihopQA are the largest gains over AutoRefine-Base, while MuSiQue is best among 3B methods.Bamboogle reaches 0.4560, second only to GiGPO’s 0.641 on its 125-sample set.Source: §4.2 · p. 5

Training dynamics

• Distillation loss decreases from ∼0.66 to ∼0.20, while query formulation becomes progressively more entity-specific and decomposed.Training remains stable through 501 steps without oscillations or divergence.Source: §4.3 · p. 5

Objective comparison

• Forward-KL achieves the highest validation EM and continues improving through 400 steps, whereas reverse-KL alternatives either collapse or plateau below it.Rev-KL Adaptive avoids collapse but reaches lower EM, 0.470 versus 0.493 for forward-KL.Source: §4.4 · p. 6, Figure 4 · p. 6, Table 2 · p. 6

Policy reshaping

• OPD expands behavioral diversity and controls policy drift, with entropy 1.35 versus 0.82 for the base model and KL ∼0.2 versus pure RL’s KL >0.8 before collapse.Average search turns decrease from 3.5 to 2.7, while forward-KL gradient norms decrease from 1.1 to 0.7 over 418 steps.Source: §4.6 · p. 6

Complementarity with RL

• OPD and RL contribute complementary capabilities: OPD establishes search behaviors, while RL refines answer accuracy.The sequential pipeline reaches 0.4402, whereas joint optimization reaches 0.3660 because of objective interference.Source: §5 · p. 6

Teacher scale and two-stage training

• OPD→RL achieves HotpotQA 0.458 and 2Wiki 0.426, exceeding pure RL by +32.1% and +35.7%, respectively.Pure RL remains competitive on single-hop QA with SH-Avg 0.514 but underperforms on multi-hop tasks.Source: §5.1 · p. 7

Offline SFT comparison

• OPD→RL reaches 0.4402, exceeding Offline SFT→RL at 0.4185 by +5.2%.Pure OPD at 0.3655 also exceeds Offline SFT at 0.3264 by +12.0% without trajectory filtering or data construction.Source: §5.1 · p. 7

Teacher scale and two-stage training

• OPD→RL reaches 0.4402, outperforming pure OPD at 0.3655 by +20.4%.The result supports using OPD as a behavioral foundation and RL as a later refinement stage.Source: §5.1 · p. 7
• The two-stage pipeline with a 7B teacher reaches 0.4205, exceeding pure OPD with the larger 14B teacher.Source: §5.1 · p. 7

• OPDSearch+ uses a frozen off-the-shelf teacher and requires no task-specific training.Source: §6 · p. 7
• The 3B OPD→RL pipeline achieves mean EM 0.4402 and outperforms all 3B baselines, with particularly strong multi-hop gains.Source: §6 · p. 7