PaperlayerView exact revision on arXiv

Source-linked AI summary

CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning

Wenjin Liu, Chenxi Wang, Jiapu Wang, Zhe Cui, Anh Tuan Luu, Haoran Luo

arXiv:2609.29518v1cs.LGcs.CE

TL;DR

RL and on-policy distillation have limits when correct reasoning is not reachable through the student's sampled trajectories. CataOPD uses catalytic guidance to elicit verified student-produced trajectories and internalize them for catalyst-free inference. It reports gains on hard problems and out-of-distribution benchmarks.

  • Problem

    RL lacks a positive correctness signal when no correct trajectory is sampled, while OPD is constrained by reasoning reachable under the student's on-policy distribution.

    Source: §1 · p. 1

  • Method

    CataOPD routes all-failed groups through self-rescue before catalytic guidance, then internalizes verified student-produced trajectories using token weights based on guided-to-unaided log-probability gaps.

    Source: §1 · p. 2, §4.3 · p. 4

  • Results

    CataOPD surpasses recent baselines, with gains concentrated on hard problems and catalyst-free out-of-distribution generalization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CataOPD internalizes student-produced correct trajectories into a policy that reasons without a teacher.

    Source: §1 · p. 2

    CataOPD requires a verifier for identifying correct student-produced trajectories, making the current formulation most suitable for tasks with reliable answer checking, such as mathematics and code.

    Source: J Limitations · p. 18

Abstract

from arXiv · show

Reinforcement learning (RL) and on-policy distillation (OPD) are two representative paradigms for improving large language model reasoning. However, when no correct trajectory is sampled, RL lacks a positive correctness signal, while OPD remains constrained by the reasoning trajectories reachable under the student's on-policy distribution. Therefore, we propose CataOPD, where the teacher acts as a catalyst rather than a target, expanding reachability while internalizing verified student-produced trajectories into a catalyst-free policy. Self-Rescue Routing uses empirically all-failed groups as routing signals rather than teacher-intervention triggers, first seeking correct trajectories through additional on-policy self-sampling. For problems unresolved after self-rescue, Catalytic-Guided Self-Resolution uses catalytic guidance to elicit a verified student-produced trajectory in the guided student distribution. Barrier-Weighted Internalization weights tokens by guided-to-unguided log-probability gaps, focusing updates on decisive tokens difficult without guidance. Experimental results show that CataOPD outperforms current baselines, extends independent student reasoning to still-unrecovered problems, and improves out-of-distribution generalization under catalyst-free inference. Our project is available at https://github.com/QwenQKing/CataOPD.

1 Introduction

The introduction identifies limits in RL and on-policy distillation when students cannot sample correct reasoning, then presents CataOPD as a way to internalize verified student-produced trajectories for catalyst-free use. Its reported gains are concentrated on hard problems and include out-of-distribution generalization.

1 Introduction

  • RL lacks a correctness-directed update when a sampled group has no correct trajectory, while OPD may miss correct reasoning outside the student's unaided distribution.Directly distilling off-support teacher trajectories can resist internalization; uniform token-level supervision may also underweight decisive positions.Source: §1 · pp. 1–2
  • CataOPD uses the teacher as a catalyst, first routing all-failed groups to additional self-sampling and escalating only unresolved problems to catalytic guidance.Both routes seek verified student-produced trajectories rather than using teacher trajectories as distillation targets.Source: §1 · p. 2
  • Barrier-Weighted Internalization weights tokens by clipped guided-to-unaided log-probability gaps to focus updates on decisive tokens that remain difficult unaided.The method targets the privilege-transfer gap between guided and unaided student policies.Source: §1 · p. 2
  • CataOPD surpasses recent baselines, with gains concentrated on hard problems and catalyst-free out-of-distribution generalization.The introduction reports that students solve problems at inference without the teacher that they would otherwise fail.Source: §1 · p. 2

2 Related Work

Related work situates CataOPD among verifier-based reinforcement learning and on-policy distillation. The latter includes both external-teacher supervision and self-distillation with privileged conditioning.

2 Related Work

  • Verifiable-reward RL uses a rule-based verifier to score outcomes, and group-relative methods include DAPO, GSPO [Zheng et al. 2025], and GCPO [Gu et al. 2026].Other cited variants modify clipping, sequence-level ratios, credit assignment, or replay.Source: §2 · p. 2
  • OPD provides token-level teacher supervision on student rollouts; external-teacher methods vary in divergence and token selection, while OPSD uses privileged conditioning of the same model.Examples include DistiLLM-2 [Ko et al. 2025], AlignDistil [Zhang et al. 2025], Self-Distilled Reasoner [Zhao et al. 2026], and OPID [Yang et al. 2026].Source: §2 · p. 2
  • Figure 3 presents CataOPD as catalytic on-policy distillation, with the teacher serving as a catalyst rather than a target.Source: Figure 3 · p. 3

3 Problem Formulation

The formulation represents a question, its verification target, and a student-generated reasoning trajectory under an autoregressive policy. It defines correctness and unaided success to support reward calculation and routing.

3 Problem Formulation

  • For question q with verification target a⋆(q), the trajectory y consists of reasoning tokens ending in a final answer and is assigned probability by autoregressive policy πθ.The trajectory probability factorizes over tokens conditioned on the generated prefix y<t.Source: §3 · p. 3
  • Correctness is determined by applying a final-answer extractor and rule-based verifier to the trajectory's answer.The binary correctness reward r(q, y) is 1 when the verifier accepts and 0 otherwise.Source: §3 · p. 3
  • The composite optimization reward combines correctness with a format reward, so all-failed groups can still have varying rewards and nonzero group-relative advantages.Uniform correctness within a group does not guarantee that its optimization advantages vanish.Source: §3 · p. 3
  • Unaided pass rate is the expected binary correctness under unaided sampling, and empirically all-failed groups are routed through self-rescue before external catalysis.Catalysis is reserved for cases that remain unresolved after self-rescue.Source: §3 · p. 3

4 Method

CataOPD uses the teacher as a catalyst, not a target, to obtain verified student-produced trajectories and internalize them into a catalyst-free policy. Its method routes problems through self-rescue, guided student resolution, and barrier-weighted training.

4.1 Self-Rescue Routing

  • Self-Rescue Routing treats all-failed groups as signals for additional on-policy sampling, using a recovered verified trajectory as the unaided policy’s internalization target.For questions with no correct initial sample, the method selects the first correct self-rescue sample; additional sampling strictly increases recovery probability when the fixed policy’s success probability is between zero and one.Source: §4.1 · pp. 3–4

4.2 Catalytic-Guided Self-Resolution

  • For questions still unresolved after self-rescue, Catalytic-Guided Self-Resolution provides method-level guidance and accepts the first verified student sample from up to H rounds.The teacher withholds the reference answer; unresolved questions are excluded from internalization.Source: §4.1 · p. 3, §4.2 · p. 4

4.3 Barrier-Weighted Internalization

  • Barrier-Weighted Internalization weights accepted student-produced correct trajectories by clipped guided-to-unaided token log-probability gaps, emphasizing tokens made more probable by guidance.Weights are 1 + βb_t; valid-token weights are mean-normalized per trajectory, and guidance enters through detached weights while likelihood updates use only the question.Source: §4.3 · p. 4

4.4 Catalytic-Guided Training

  • Catalytic-Guided Training combines accepted self-rescue and catalytic samples, optimizing internalization alongside clipped policy and KL terms while conditioning likelihood updates only on the question.Guidance affects internalization only through detached barrier weights.Source: §4.4 · p. 4

5 Experiments and Results

CataOPD is evaluated across twelve reasoning benchmarks, with results covering overall and OOD accuracy, hard-problem solvability, component contributions, training-time recovery, and guidance internalization. Across evaluated students, it shows broad catalyst-free gains, including on held-out benchmarks and hard problems.

5.1 Experimental Setup

  • The evaluation covers twelve benchmarks—eight in-distribution and four held-out OOD—and uses accuracy, Pass@k, retention, recovery, and mechanism diagnostics.Students are Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B; GPT-5.4-mini serves as the catalytic teacher during training.Source: §5.1 · p. 5

5.2 Main Results (RQ1)

  • CataOPD attains the highest same-backbone average accuracy, exceeds GPT-5.4-mini and the baselines, and ranks first across backbone–dataset pairs.The reported gains hold across all three student backbones and eight benchmarks under catalyst-free inference.Source: §5.2 · pp. 5–6

5.3 OOD Generalization (RQ2)

  • CataOPD achieves the strongest same-backbone OOD average for every evaluated student, leads on all four OOD benchmarks, and does so without a catalyst at inference.Figure 4 reports per-benchmark accuracy, the four-benchmark average, and comparison with the strongest baseline.Source: Figure 4 · p. 6, §5.3 · p. 6

5.4 Hard-Problem Analysis (RQ3)

  • Blind and Hard gains exceed Overall gains for both analyzed backbones, while Pass@8 newly solved cases far outnumber regressions and nearly all base-solved cases remain solvable.Figure 5 summarizes gains with 95% confidence intervals and Pass@8 solvability transitions.Source: Figure 5 · p. 6, §5.4 · p. 6

5.5 Component Ablation (RQ4)

  • Removing Self-Rescue Routing causes the largest performance decline on every benchmark; removing guidance or barrier weighting also lowers performance on all eight benchmarks.Barrier weighting ranks second on average, while guidance provides a positive gain across the evaluated benchmarks.Source: Table 2 · p. 7, §5.5 · p. 7

5.6 Training Dynamics (RQ5)

  • After-recovery verified-target coverage exceeds initial-rollout coverage in every logged window for all three students.The positive gap holds for Qwen3-1.7B, Qwen2.5-3B, and Qwen2.5-7B despite differences in initial coverage and gain magnitude.Source: Figure 6 · p. 7, §5.6 · p. 7

5.7 Mechanism Diagnostics (RQ6)

  • Across 200 successfully catalytically recovered Qwen2.5-3B trajectories, matched guidance raises verified-trajectory likelihood, unaided gains covary positively with guided gains, and barrier mass concentrates on tokens with the largest likelihood gaps.The comparison includes unaided and shuffled guidance conditions.Source: Figure 7 · p. 7, §5.7 · p. 7

6 Conclusion

CataOPD combines self-rescue, guided recovery, and barrier-weighted internalization to train catalyst-free mathematical reasoning. The appendix analysis shows that additional self-sampling can recover verified student trajectories and provide a correctness-directed update.

6 Conclusion

  • CataOPD combines Self-Rescue Routing, Catalytic-Guided Self-Resolution, and Barrier-Weighted Internalization, and reports best average accuracy per backbone with consistent OOD and hard-problem gains.The conclusion also reports retaining nearly all base-solved cases under Pass@8 on both analyzed backbones.Source: §6 · p. 7

A.1 Proof of Proposition 1

  • An all-failed correctness event does not imply equal composite rewards or zero GRPO advantages; a recovered verified trajectory nevertheless supplies a correctness-directed logit gradient.At valid positions, the gradient is nonzero unless the target token already has probability one.Source: §A.1 · p. 10
  • When unaided success probability is between zero and one, increasing the self-rescue sampling budget exponentially decreases the probability of missing every correct trajectory.A recovered correct student-produced trajectory supplies a verified target for unaided internalization.Source: §A.1 · p. 10

A.2 Proof of Proposition 2

CataOPD retries unresolved questions with guided student sampling to obtain verified student-produced trajectories, then internalizes them under the unaided policy. Its training procedure combines all-failed-group routing, bounded catalytic retries, and token-level barrier weighting.

A.2 Proof of Proposition 2

  • A.2’s H-round guided retry reaches the verified target set with positive probability if any reachable round has positive success probability, and accepted targets are student-produced.The accepted trajectory is added for internalization rather than using a teacher-generated trajectory.Source: §A.2 · p. 11

A.3 Proof of Proposition 3

  • Barrier weighting raises direct first-order unaided update coefficients for above-average-barrier tokens, but does not order total parameter-coupled changes.Normalized weights preserve the trajectory’s average loss scale; guidance affects the weights while the likelihood is conditioned only on the question.Source: §A.3 · pp. 11–12

B Catalytic Prompt Details

  • The shared student prompt specifies reasoning and answer tags and stays unchanged at catalyst-free inference; training-only teacher guidance supplies a method without revealing the answer.Self-rescue and teacher guidance are training-time mechanisms only.Source: B Catalytic Prompt Details · p. 12, Figure 8 · p. 12

C CataOPD Algorithm Details

  • The algorithm routes empirically all-failed groups to additional unguided sampling, escalates only unresolved questions to at most H guided rounds, and internalizes verified student trajectories.Its full objective combines clipped on-policy updates over initial groups, KL regularization, and barrier-weighted internalization.Source: C CataOPD Algorithm Details · pp. 12–13

D Dataset Details

The evaluation uses twelve public mathematical-reasoning datasets spanning training and held-out sources, with disjoint training, validation, and evaluation splits.

D Dataset Details

  • The twelve datasets cover word problems, algebra, geometry, number theory, competition mathematics, and general mathematical reasoning.Examples include MAWPS for arithmetic and algebraic word problems and Omni-MATH for olympiad-level mathematics.Source: D Dataset Details · pp. 13–14
  • The first eight datasets serve as training and same-source evaluation sources, while the last four are held-out sources for catalyst-free evaluation.The held-out sources are not sampled as named training sources.Source: D Dataset Details · p. 14
  • The deduplicated sampling yields 10,240 training, 1,536 validation, and 1,536 evaluation instances with no exact question overlap across splits.Each training source contributes 1,280 training instances; each of the twelve sources contributes disjoint sets of 128 validation and 128 evaluation instances.Source: D Dataset Details · p. 14

E Baseline Details

The comparison includes seven primary training baselines across supervised fine-tuning, verifiable-reward reinforcement learning, teacher-guided distillation, and on-policy self-distillation, alongside reference models.

E Baseline Details

  • The seven primary training baselines span supervised fine-tuning, reinforcement learning from verifiable rewards, black-box teacher-guided OPD, and on-policy self-distillation.GPT-5.4-mini CoT-SFT is included as an offline-distillation reference, alongside Base LLM and GPT-5.4-mini reference points.Source: E Baseline Details · p. 14
  • GRPO uses verifier-based composite rewards normalized within sampled groups and updates the policy with a clipped objective.Its composite reward combines answer correctness with format compliance and lies in [−1, 1].Source: E Baseline Details · p. 14
  • The on-policy distillation baselines include OPSD, which distills privileged-conditioned teacher distributions into a question-only policy, and OPCD, which uses a context-conditioned self-teacher.RLCSD contrasts teacher-student gaps under correct and incorrect hints, while SDPO distills successful rollouts through a reprompted self-teacher.Source: E Baseline Details · p. 14

F Evaluation Metrics

Evaluation covers single-sample accuracy, multi-sample solvability, retention of base-solved questions, all-failed-group recovery, and diagnostics of guided and unaided token likelihoods.

F Evaluation Metrics

  • Pass@8 measures whether at least one of eight independently sampled trajectories is correct, while accuracy measures single-sample verifier correctness.The same rule-based verifier is used for evaluation correctness and reward computation.Source: F Evaluation Metrics · pp. 14–15
  • Solved-problem retention is 99.8%/99.9% for Qwen2.5-3B/7B under the stated protocol.The corresponding regression rate is 1 − Ret.Source: F Evaluation Metrics · p. 15
  • All-failed-group recovery measures whether self-rescue or catalytic resolution obtains a verified target after every trajectory in the initial group fails.The method reports this over training windows with at least one all-failed group.Source: F Evaluation Metrics · p. 15
  • Likelihood diagnostics compare guided and unaided contexts and relate trajectory-mean gains using Pearson and Spearman correlations; barrier concentration tracks mass in the top token fractions.The reported concentration fractions use α values 0.1, 0.2, and 0.3.Source: F Evaluation Metrics · p. 15

G Implementation Details

The implementation uses three student backbones and a training-time catalytic teacher, with evaluation performed without teacher guidance. Routing analyses describe how initially all-failed groups are recovered through self-rescue and successive guided rounds.

H.1 Routing and Teacher Budget

  • Table 4 reports routing and teacher budgets across three backbones, distinguishing initially all-failed groups, teacher-attempted groups, and total recovery; teacher token counts exclude student rollout tokens.Confidence intervals are 95% step-cluster bootstrap intervals.Source: Table 4 · p. 16, §H.1 · p. 17
  • Figure 9 separates initially all-failed groups into self-rescued, teacher-recovered, and unrecovered outcomes, and reports cumulative and per-round recovery as a student-first pathway rather than a task-accuracy metric.The decomposition covers successive guided rounds R1–R5.Source: Figure 9 · p. 16, §H.1 · p. 17

G Implementation Details

  • Experiments use Qwen3-1.7B as the student with GPT-5.4-mini as the training-time catalytic teacher; at evaluation, all methods use the same guidance-free solving prompt and rule-based verifier, without teacher calls or guidance.All trainable methods use the same data splits.Source: G Implementation Details · pp. 16–17

H.2 Self-Rescue Ablation Dynamics

Self-rescue improves recovery from initially all-failed groups while reducing teacher budget per verified target in matched runs. Mechanism analyses and case studies describe conditional recovery behavior, alongside practical limitations and planned extensions.

H.2 Self-Rescue Ablation Dynamics

  • 35.97% of initially all-failed groups yield verified targets with Full CataOPD versus 23.75% without Self-Rescue; teacher calls and input tokens per target are 10.1 and 16.2K versus 17.6 and 33.1K.The comparison uses matched Qwen2.5-3B runs over all 80 training steps.Source: §H.2 · p. 17
  • Across every displayed 10-step window, Full CataOPD has higher all-failed-group recovery and fewer teacher calls and input tokens per verified target than the w/o Self-Rescue variant.The matched-window pattern is consistent with student-first routing before external catalysis.Source: §H.2 · pp. 17–18

H.3 Mechanism Analysis Protocol and Robustness

  • Mechanism diagnostics use 200 fixed, eligible Qwen2.5-3B trajectories and paired likelihood comparisons across policy and context conditions.Eligibility requires five failed initial samples and five failed self-rescue samples, followed by a verified student trajectory within at most five catalytic rounds. The analysis characterizes successfully recovered trajectories, not population-level accuracy or an unconditional causal effect.Source: §H.3 · p. 18

I Case Study

  • In the Figure 12 word-problem case, CataOPD returns the correct answer 18 while all seven comparison systems shown return different final answers.The case requires separate game calculations and correct aggregation of the resulting quantities.Source: I Case Study · p. 18, Figure 12 · p. 19
  • In the Figure 13 recurrence case, the second guided retry verifies the repeated block and returns answer 9 after five failed initial rollouts and unsuccessful self-rescue.The first guided retry corrects a local calculation but retains the incorrect period assumption.Source: I Case Study · p. 18, Figure 13 · p. 20

J Limitations

  • CataOPD currently requires verifiable rewards, reliable catalytic guidance, and training-time sampling efficiency; answer-checkable tasks such as mathematics and code fit the stated verifier requirement.Applying it to open-ended reasoning or subjective evaluation would require additional reward design.Source: J Limitations · p. 18

K Future Work

  • Future work proposes evaluation on code, symbolic, and tool-assisted reasoning, adaptive guidance allocation, and scaling to larger models and longer reasoning tasks under practical budgets.The proposed evaluation includes execution-based verification for structured tasks and learned or hybrid verifiers for less constrained tasks.Source: K Future Work · pp. 18–19
Loading 2609.29518v1…