On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

1 INTRODUCTION

• Post-training methods differ in whether trajectories are model-generated and whether supervision is dense, while external supervision persists across SFT, RLVR, OPD, and OPSD.The passage describes reference solutions, gold answers, stronger teachers, and privileged context as their respective external signals.Source: §1 · p. 1
• Scaling human-curated supervision is costly, and prior synthetic-supervision methods still rely on human-written problems that determine the training distribution.The cited examples include about 800,000 human step labels for one mathematics corpus and instruction datasets involving over 13,000 volunteers.Source: §1 · p. 1
• Earlier self-training methods filter solutions but still use curated problem sets, motivating the question of whether models can generate their own training problems.The paper identifies topic descriptors as the only required external input for generating those problems.Source: §1 · pp. 1–2
• Seeded only by coarse topic descriptors, OPST generates and filters its own questions and privileged information, then trains through on-policy self-distillation.Pseudo-verification checks whether a problem is well-formed; a consensus gate discards problems the student already consistently solves.Source: §1 · p. 2, Figure 1 · p. 2
• The teacher uses the generated answer as privileged context to produce reasoning supervision, while the student receives only the question; an EMA teacher stabilizes iterative training.The paper argues this lets the teacher fill in reasoning steps omitted from the generated answer.Source: §1 · p. 2
• OPST improves language and multimodal reasoning; on Gemma 4 E2B, it increases MMLU accuracy by +10.2 points and Visual OPST increases VQAv2 accuracy by +6.9 points.Performance on domains outside self-training either remains stable or also improves.Source: §1 · p. 2

2 RELATED WORK

• OPSD uses a shared model with asymmetric conditioning and a verified reference solution; later work varies privileged context or addresses leakage, instability, and agentic tasks.Examples include expert demonstrations, privileged observations, and environmental feedback.Source: §2 · pp. 2–3
• OPST removes the supplied problem-distribution assumption by generating queries from topic descriptors, using generated answers and rationales as dense token-level teacher context.Related work includes self-play with external corpus retrieval, whereas OPST generates problems from parametric knowledge alone.Source: §2 · p. 3
• Prior self-training filters solutions by reference-answer agreement or self-consistency, while label-free reinforcement learning uses majority votes or model confidence; these approaches retain curated problems.Self-play can generate tasks from topic descriptions, but its supervision is reduced to scalar rewards rather than dense token-level targets.Source: §2 · p. 3

3.1 PROBLEM FORMULATION: DATA-FREE POST-TRAINING

• Data-free post-training uses coarse topic descriptors to generate a synthetic dataset of queries and privileged information, while aiming to improve seeded domains and retain general reasoning elsewhere.The privileged information consists of a rationale and candidate answer.Source: §3.1 · pp. 3–4

3.2 PRELIMINARIES: ON-POLICY SELF-DISTILLATION

• On-policy self-distillation trains the student to match a teacher that evaluates the student’s own response prefix while receiving privileged information the student does not see.The teacher receives no gradient; existing OPSD methods typically use curated problems and a reference solution as privileged information.Source: §3.2 · p. 4

3.3 TOPIC-SEEDED CURRICULUM GENERATION

• From each topic descriptor, the generator produces a query and privileged information containing a rationale and candidate answer; the student sees only the query.The generator may be the EMA teacher or a frozen copy of the initial model, while student rollouts determine which samples are retained.Source: §3.3 · p. 4

3.4 PSEUDO-VERIFICATION & CONSENSUS GATE

• Pseudo-verification retains samples whose generated answers survive fresh-context solving and question paraphrasing, while the consensus gate discards problems the student already solves reliably.Together, the filters target well-posed samples at the student’s capability frontier, making curriculum difficulty depend on the current student.Source: §3.4 · p. 5

3.5 MODEL TRAINING

• Training updates the student by on-policy self-distillation, and the EMA teacher averages student weights to stabilize dense supervision and is the model evaluated.The method uses a bounded skewed Jensen–Shannon divergence, with α = 0.8 favoring the mode-seeking regime.Source: §3.5 · p. 5

3.6 ITERATIVE SELF-TRAINING

• OPST’s iterative loop lets student and teacher co-evolve, with a stronger teacher supplying better token-level targets, generated data, and verification in later rounds.Only improvements that persist across batches enter this cycle.Source: §3.6 · p. 6

3.7 VISUAL OPST

• Visual OPST generates image-based questions with privileged information such as detailed scene captions or detected, bounding-box-grounded spatial relations.The teacher can use the caption, the spatial relations, or both; visual verification checks candidate answers across augmented image views.Source: §3.7 · p. 6

4.1 EXPERIMENTAL SETUP

• The evaluation measures in-domain self-improvement, out-of-domain retention, and transfer to multimodal inputs using Gemma 4 models and four in-domain benchmarks.The setup also uses three OOD benchmarks; its comparisons include training algorithms on identical self-generated data and methods spanning different supervision levels.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 RESULTS

• OPST exceeds SFT on identical self-generated data by 5.6/1.3 points (2B/4B) without thinking and 2.7/3.8 points with thinking.On Gemma 4 E2B without thinking, it improves MMLU by +10.2 points, GPQA by +4.0, and HellaSwag by +7.5.Source: Table 1 · p. 7, §4.2 · p. 7
• Among methods without reference solutions, OPST has the best average in all four settings, exceeding u-OPSD by 1.0–2.8 points and R-Zero by 1.4–5.2 points.At 2B without thinking, it recovers 88% of the supervised OPSD gain (+7.3 of +8.3); with thinking, it matches supervised OPSD at both sizes (+0.2 and +0.7 points).Source: §4.2 · p. 7, Table 2 · p. 8
• Combining all three topic domains raises the in-domain average by 7.6 points, while the out-of-domain average rises by 0.8 and MATH-500 falls by 4.9.MMLU plus GSM8K topics improve MMLU by 11.3 points versus 10.2 with MMLU topics alone and yield the best out-of-domain average (+3.9); MMLU also improves by 5.1 without MMLU topics.Source: §4.2 · pp. 7–8, Table 3 · p. 8

4.3 MULTIMODAL SELF-TRAINING: VISUAL OPST

• At 2B, Visual OPST improves VQAv2 by 6.6 points with spatial QA and 4.9 with scene captions; combining them gives the largest VQAv2 gain (+6.9).At 4B, spatial QA alone performs best on VQAv2 and POPE, so the two privileged-information forms are not consistently complementary.Source: §4.3 · pp. 8–9, Table 4 · p. 9

• Without filtering, OPST improves over the base model by +5.2 points; pseudo-verification adds +2.7 and the consensus gate adds another +2.3.The filters are added incrementally, starting with OPST on all generated samples.Source: §5 · p. 9
• Removing privileged information reduces the gain over base from 10.2 to 3.1 points, while rationale-only yields 8.3 and answer-only 7.1.A placebo from a different problem on the same topic yields 2.7 points; the authors attribute the gain to sample-specific privileged information, with rationale carrying more signal than the answer.Source: §5 · p. 9
• With 75% wrong candidate answers, OPST loses 1.9 points but stays 3.3 above base, whereas SFT loses 5.7 and falls 4.4 below base.The authors report that conditioning the teacher on the generated answer, rather than imitating it, limits the influence of incorrect generations.Source: §5 · p. 9

6 CONCLUSION

• OPST uses self-generated problems and privileged information, seeded only by coarse topic descriptors, and outperforms SFT, GRPO, and methods relying on curated questions.The conclusion also reports extension to multiple domains and vision-language models.Source: §6 · p. 9
• Pseudo-verification measures self-consistency rather than correctness, so consistent errors can enter training; experiments use hand-chosen topic descriptors and small models.Source: §6 · p. 10

AI USE STATEMENT

• The authors used generative AI for training infrastructure, experiment scripts, experimental-design feedback, and manuscript drafting and editing, but not for core research ideas or hypotheses.Source: AI USE STATEMENT · p. 10

• Generated privileged information can omit important reasoning or contain partly or wholly incorrect answers, so direct training on it inherits these defects.Source: §A.1 · p. 14
• Conditioning the teacher on generated privileged information supplies next-token distributions at every state visited by the student, rather than supervision only over generated tokens.The teacher uses its own knowledge under exposure to that context, completing omitted reasoning and tempering the influence of errors according to what it finds plausible.Source: §A.1 · p. 14

A.2 FURTHER ABLATIONS

• 47.7 is the reported MMLU result with EMA rate β = 0.05; without EMA, the model reaches 38.1 versus the base model’s 37.5, while β = 0.01 reaches 45.2.The reported ablations identify β = 0.05 as best; excessive smoothing slows progress.Source: Table 6 · p. 14, §A.2 · p. 15
• Accuracy increases through 285 samples per round, then saturates and decreases slightly to 46.9 at 570 samples on MMLU.Larger rounds add generation cost without improving results.Source: Table 6 · p. 14, §A.2 · p. 15
• +5.6 emerges after the first three rounds, and MMLU accuracy peaks at 47.7 after seven rounds; further rounds add no improvement, while performance remains around 46.Source: Table 6 · p. 14, §A.2 · p. 15
• The EMA teacher is 1.8 points more accurate than the student at the end of training on MMLU.The authors attribute this consistency with removal of noise from recent student batches through averaging.Source: Table 6 · p. 14, §A.2 · p. 15

A.3 VARIANCE ACROSS SEEDS

• Across three seeds, standard deviations stay at or below 1.2 points, below OPST’s gains over the base model.With thinking, variance is highest on GPQA and GSM8K and lowest on MMLU (0.1); on GPQA Diamond, one question corresponds to about 0.5 points.Source: §A.3 · p. 15, Table 7 · p. 15

A.4 GENERALIZATION TO QWEN

• 6.0 points: OPST raises Qwen3-4B-Base average accuracy from 66.7 to 72.7, improving all four benchmarks, with its largest gain on GSM8K (+13.3).The other gains are GPQA (+5.1), MMLU (+3.1), and HellaSwag (+2.5); multiple-choice benchmarks are scored by log-likelihood over answer options.Source: §A.4 · p. 15

A.5 OPST ALGORITHM

• Each OPST round synthesizes and filters problems with privileged information, trains the student on its own rollouts against a teacher initialized from the same weights, and returns the EMA teacher.After each update, the teacher tracks the student through an exponential moving average with rate β.Source: §A.5 · pp. 15–16

A.6 EVALUATION PROTOCOL

• 0% parsing failures and invalid answers were observed before and after training; evaluation uses closed-option log-likelihood for multiple choice and exact-match or symbolic grading for generated answers.The protocol uses 5-shot MMLU, zero-shot remaining multiple-choice tasks, and 8-shot chain-of-thought GSM8K.Source: §A.6 · p. 16

A.7 OPST PROMPT TEMPLATES

• Text synthesis prompts generate questions with rationales and candidate answers from topic descriptors, while students receive only questions and teachers receive privileged context.Multiple-choice prompts target MMLU, GPQA, and HellaSwag; open-ended prompts target GSM8K.Source: §A.7 · pp. 16–18

A.8 VISUAL OPST PROMPT TEMPLATES

• Visual OPST synthesizes image-grounded questions, builds privileged context from scene captions or detected-object relations, verifies answers across augmented views, and gives that context only to the teacher.Object counting has a dedicated synthesis prompt; other visual topics share a template.Source: §A.8 · pp. 19–20