PaperlayerView exact revision on arXiv

Source-linked AI summary

Ask to Distill: Learning to Synthesize Data for On-Policy Distillation

Jiashuo Sun, Hao Zhang, Xueqiang Xu, Heng Wang, Yixuan Xie, Pengcheng Jiang, Zhenrui Yue, Jiajun Fan, Peiran Li, Ge Liu, Shaowen Wang, Jiawei Han

arXiv:2610.27450v1cs.AIcs.CL

TL;DR

Questions determine which student-generated trajectories receive teacher supervision, but their distillation value is difficult to assess without reference answers. Ask-to-Distill optimizes synthetic questions using a frozen student–teacher pair. It exceeds real-bank OPD on average across evaluated pairs, and optimized questions can elicit denser signals from shorter responses.

  • Problem

    A newly generated OPD question’s value for a target student–teacher pair must be assessed without access to its correct answer.

    Source: §1 · p. 2

  • Method

    A2D optimizes a student-initialized question synthesizer against a frozen student–teacher pair using support-aware distributional signals and length and diversity controls.

    Source: §1 · p. 2

  • Results

    A2D exceeds real-bank OPD on the average scores of evaluated student–teacher pairs, with improvement magnitude varying substantially across pairs.

    Source: §1 · p. 2, §3.4 · p. 11

  • Takeaways & Limitations

    Optimized questions can elicit denser distributional signals from shorter student responses.

    Source: §1 · p. 2

    The reward is estimated from one student response per question, and the synthesizer is optimized once for a fixed student–teacher pair.

    Source: §5 · p. 17

Abstract

from arXiv · show

On-policy distillation (OPD) can succeed with surprisingly few training questions, motivating a view of questions as carriers of distributional knowledge transfer. How can we assess whether a newly generated question is valuable for distillation, and learn to generate better questions---without access to its correct answer? We introduce Ask-to-Distill (\method), a framework that treats synthetic questions as distribution elicitors and optimizes the teacher supervision they expose along student-generated trajectories. A student-initialized synthesizer is optimized using a distributional surrogate for question value, computed from a frozen student--teacher pair without reference answers, correctness rewards, or per-candidate student updates. The reward combines teacher--student disagreement with teacher probability mass on the student's candidate support, together with length and diversity controls. The generated questions are filtered and deduplicated into a prompt-only bank for standard OPD. Across mathematical reasoning, code generation, and agentic tasks, our method improves average performance and teacher-gap recovery over unoptimized student-generated prompts and KL-filtered real questions, while matching or exceeding real-bank OPD in the evaluated settings. Signal analyses further show that optimized questions can elicit denser supervision and more total signal from shorter student responses, supporting elicited distributional supervision as an objective for synthetic distillation data. Our code, data and models are publicly available at https://github.com/GasolSun36/Ask_to_Distill.

1. Introduction

OPD questions shape which student-generated trajectories receive teacher feedback, so their value depends on the supervision elicited for a particular student–teacher pair. Ask-to-Distill optimizes synthetic questions using that elicited signal and reports higher average scores than real-bank OPD across evaluated pairs.

1. Introduction

  • Questions determine which student-generated trajectories receive teacher feedback, making elicited supervision—not only question content—central to distillation data.OPD provides token-level teacher guidance on student-visited states without a task-specific correctness verifier.Source: §1 · pp. 1–2
  • A2D trains a student-initialized synthesizer with a frozen student–teacher pair, then filters and deduplicates generated questions for standard OPD.Its reward combines support-aware teacher–student signals with response-length and question-diversity controls.Source: §1 · p. 2
  • A2D exceeds real-bank OPD on the average scores of the evaluated student–teacher pairs, while analyses show denser signals from shorter responses.The framework evaluates mathematical reasoning, code generation, and agentic tasks.Source: §1 · p. 2
  • A2D scores question value from distributional supervision without reference answers, task-specific correctness rewards, or per-candidate student updates.The score combines support-aware teacher–student signals while controlling response length and question diversity.Source: §1 · p. 2

2. Method

The method treats synthetic questions as elicitors of student–teacher supervision and optimizes a student-initialized question generator against a frozen pair. It curates the generated prompts and applies candidate-restricted OPD to the original student.

2.1. Problem formulation: synthetic prompts as distribution elicitors

  • Synthetic questions are prompt-only: the student generates trajectories from them and receives teacher token-level supervision without reference responses.Question value depends on the supervision exposed for a particular student–teacher pair.Source: §2.1 · p. 3

2.2. Overview

  • The pipeline optimizes questions, curates a prompt bank, then performs OPD on the original student; the teacher remains frozen throughout.The synthesizer and scoring student share initial parameters but are separate copies, and only the synthesizer is updated during question optimization.Source: §2.2 · p. 4

2.3. The learning-value reward

  • A2D scores questions by combining teacher–student discrepancy with teacher probability mass on the student’s candidate support, favoring positions where both are present.The resulting score is a proxy for useful supervision, not a guarantee of learning gain.Source: §2.3 · p. 4
  • The reward aggregates token signals with sublinear length normalization, a response-budget penalty, and a batch-level diversity discount for similar questions.Length controls reduce, but do not eliminate, the incentive for longer trajectories; similarity clusters determine diversity discounts.Source: §2.3 · p. 5

2.4. Optimizing the synthetic data distribution

  • A2D initializes its synthesizer from the student and uses GRPO to optimize question-level rewards computed from frozen-pair responses, without answer labels or per-candidate student updates.Student responses score generated questions; the update applies to the question-generating policy.Source: §2.4 · p. 5

2.5. Synthetic data curation

  • Curation filters malformed or incomplete outputs, removes lexical and semantic duplicates, and retains questions in a prompt-only bank for downstream OPD.A frozen embedding model is used only for post-generation semantic deduplication; stored placeholder answers are ignored.Source: §2.5 · pp. 5–6

2.6. On-policy distillation

  • Downstream OPD uses student top-K candidates in a PPO-style clipped update, a reverse-KL-style signal that is not exact full-vocabulary reverse-KL minimization.The candidate sets and advantages are computed from the rollout policy and held fixed during the update.Source: §2.6 · p. 7

3. Experiments

The experiments evaluate A2D across mathematical reasoning, code generation, and agentic tasks, and examine how elicited-divergence scores and support-aware signal relate to learning gains, downstream accuracy, and question characteristics.

3.4. Main Results

  • A2D achieves the highest average among compared distillation methods in all six mathematical and code settings.For math, A2D scores 46.2, 51.6, and 64.5 versus Vanilla OPD’s 43.0, 50.5, and 62.9; for code, the corresponding averages are 39.6, 35.7, and 21.2 versus 34.5, 30.2, and 20.2. It is best or tied-best in 16 of 18 code benchmark–pair combinations, though gains and teacher-gap recovery vary across pairs.Source: Table 2 · p. 8, §3.4 · pp. 10–11

3.5. Ablation: Can a stronger model replace synthetic data optimization?

  • With RL, student- and teacher-initialized synthesizers both improve, but neither initialization consistently dominates across the three code benchmarks.The student-initialized RL variant reaches roughly 35% on LCB, above the 33.7% teacher reference, but remains below teacher references on APPS-Intro and APPS-Interview; checkpoint curves are descriptive, not uncertainty estimates.Source: Figure 3 · p. 9, §3.5 · pp. 11–12

3.6. Ablation: which reward components matter?

  • Removing compatibility, diversity, or the length penalty reduces accuracy on all three AIME benchmarks, with drops ranging from 2.1 to 6.6 percentage points.The full reward reaches 45.0%, 35.8%, and 29.2% on AIME24, AIME25, and AIME26; the ablation evaluates downstream accuracy rather than synthesizer reward.Source: Figure 4 · p. 9, §3.6 · p. 12

3.7. Applicability to agentic tasks

  • On τ 2-Bench, A2D averages 41.9%, exceeding Vanilla OPD’s 40.4% and KL-Filtered OPD’s 40.9%, and raises teacher-gap recovery from 79.2% to 95.2%.Its Airline, Retail, and Telecom scores are 46.3%, 49.4%, and 30.0%; the average remains 1.4 percentage points below the 43.3% teacher reference.Source: Table 3 · p. 9, §3.7 · pp. 12–13

3.9. Training-token efficiency and checkpoint gains

  • A2D’s best accuracy exceeds Vanilla OPD by 2.53 points on R1-Distill-1.5B and 2.10 points on R1-Distill-7B, without a substantial increase in downstream training tokens.The 7B model reaches its best checkpoint with substantially fewer training tokens; these results do not establish lower end-to-end cost because synthetic-question generation and scoring also incur costs.Source: Figure 5 · p. 11, §3.9 · p. 13

3.10. Iterative Training

  • In the evaluated code setting, two additional optimization rounds add only 0.44 points over round one, despite extra question-generation and distillation effort.Average accuracy rises from 29.03% to 40.53% in round one, then reaches 40.82% and 40.97% in rounds two and three; the comparison uses each round’s highest observed checkpoint average, an optimistic assessment rather than a validation-selected final result.Source: §3.10 · pp. 13–14

3.8. Cross-Domain Transfer between Math and Code

  • A2D outperforms real source-domain questions on every reported target benchmark in both code-to-math and math-to-code transfer.Code-to-math average accuracy rises from 31.38% to 38.60%, versus a 5.33-point gain for Vanilla OPD; math-to-code rises from 47.00% to 52.40%, versus Vanilla OPD’s 51.13%. No additional target-domain training is used.Source: §3.8 · p. 13, Figure 7 · p. 14

3.11. Case study: where does the teacher provide useful signal?

  • A synthetic question elicits denser supervision than a real question despite a simpler surface form, with a shorter rollout and higher mean support-aware score.The synthetic rollout has 1,241 tokens versus 2,002 and mean support-aware score 0.172 versus 0.117; both student answers are incorrect.Source: Figure 6 · p. 12, §3.11 · p. 14
  • In the examples, support-aware scores rank positions almost identically to discrepancy (Spearman ρ = 0.994 and 0.997), while surface complexity alone incompletely indicates question value.The synthetic question’s cumulative score remains slightly lower than the real question’s (214 versus 234), despite retaining approximately 91% of scored signal with 62% of response tokens.Source: §3.11 · p. 14
  • Synthetic questions produce 1,020.7 versus 1,280.2 response tokens and cumulative support-aware scores of 181.7 versus 141.7, with higher mean token-level scores across all displayed response-position bins.These samples show a 20% response-length reduction and a 28% increase in cumulative signal.Source: §3.11 · p. 15, Figure 8 · p. 15

3.12. Is Elicited Divergence a Learning Signal?

  • Mean divergence predicts per-question learning value: its correlation with probe KL decrease is ρ = 0.727, and the highest-score quintile yields 2.15× the lowest quintile’s decrease.The correlation remains positive after controlling for response length, student pass rate, and student entropy (partial ρ = 0.386); the full score’s raw and partial correlations are 0.750 and 0.399.Source: §3.12 · p. 15
  • Higher-scoring question banks generally improve final accuracy across six benchmarks under a shared token budget, though the trend is not strictly monotonic.Average accuracy rises from 41.06% in R1 to 43.49% in R4; R5 reaches 43.11%, exceeding R1 by 2.05 percentage points.Source: §3.12 · p. 15

4. Related Work

Prior OPD work examines how student-generated trajectories and selective token supervision affect distillation. A2D differs by learning which synthetic questions elicit useful supervision for a target student–teacher pair.

4.1. On-Policy Distillation and Selective Supervision

  • Across 1,500 questions, probe KL decrease increased across quintiles ranked by mean score; the highest-scoring quintile yielded 2.15× the decrease of the lowest.Source: §4.1 · p. 16
  • Token-selection methods distinguish locally learnable disagreement from mismatch outside the student’s candidate support, while other approaches alter trajectories or teacher targets.A2D instead changes the synthetic prompt distribution while holding downstream OPD fixed.Source: §4.1 · p. 16

4.2. Synthetic Data Generation and Selection for Distillation

  • Synthetic-data work ranges from instruction-response generation and solver-feedback training to prompt selection for OPD, where question utility depends on the student–teacher pair and target capability.The cited studies report that small or self-generated question sets can be effective, but targeted selection does not consistently beat random sampling.Source: §4.2 · p. 16
  • A2D trains a synthesizer to shape questions using support-aware supervision rather than surface complexity, answer correctness, or raw disagreement alone.It reuses the target student and teacher, then curates a prompt-only bank for OPD; a frozen embedding model is used only for semantic deduplication.Source: §4.2 · pp. 16–17

5. Conclusion

A2D treats synthetic questions as distribution elicitors and optimizes them to expose useful teacher supervision along student-generated trajectories. Across the evaluated task domains, it improves average performance and teacher-gap recovery over selected alternatives, while its reward and optimization remain bounded to a fixed student–teacher pair.

5. Conclusion

  • Across mathematical reasoning, code generation, and agentic tasks, A2D improves average performance and teacher-gap recovery over unoptimized student-generated prompts and KL-filtered real questions, while matching or exceeding real-bank OPD in evaluated settings.Signal analyses also show denser supervision from shorter student responses.Source: §5 · p. 17
  • A2D trains a student-initialized synthesizer against a frozen student–teacher pair, combining disagreement with teacher probability mass on student candidate support and length and diversity controls.Generated questions are filtered and deduplicated into a prompt-only bank for standard OPD.Source: §5 · p. 17
  • The learning-value reward is estimated from one student response per question, and the synthesizer is optimized once for a fixed student–teacher pair.Adaptive generation as the student improves and combining question generation with token-level selection remain future directions.Source: §5 · p. 17

A. Experimental Details

The appendix identifies Algorithm 1 as a summary of the method.

A. Experimental Details

  • Algorithm 1 summarizes the method.The supplied passage does not specify the algorithm’s steps.Source: §A.1 · p. 21

A.2. Implementation Details

The implementation details specify synthesizer training, question-bank construction, downstream OPD, and checkpoint selection. Each method uses one checkpoint selected by a designated benchmark rather than selecting separately for every benchmark.

A.2. Implementation Details

  • The framework is based on verl, and GRPO trains the synthesizer for 60 steps with 8 prompts and group size 8, yielding 64 candidate questions per step.The learning rate is 1 × 10^-6; checkpoints are saved every 5 steps.Source: §A.2 · p. 21
  • Reward shaping uses α = 0.5, λlen = 0.3, λdiv = 1.0, and clustering threshold τ = 0.1.The discrepancy scale δref is the 95th percentile of δt estimated from 200 calibration questions; ϵ = 10^-8.Source: §A.2 · p. 21
  • The synthesizer generates 20,000 raw questions in 10 shards, then applies Jaccard and cosine deduplication thresholds of 0.7 and 0.95.Question-generation budget is 4096 tokens; student-response budget is 7168 tokens.Source: §A.2 · p. 21
  • Downstream OPD uses top-K student candidates with K = 16, samples 4 responses per prompt, and trains for 2 epochs with batch size 64.Training uses a learning rate of 1 × 10^-6 and fp32 parameters.Source: §A.2 · p. 21
  • For each student–teacher pair and OPD method, one checkpoint is selected using AMC23 for math or LiveCodeBench for code, and all benchmark scores use that checkpoint.Those checkpoint-selection benchmark scores are not held-out estimates.Source: §A.2 · p. 21

A.3. Evaluation Details

The evaluation details specify how mathematical answers and generated code are judged, alongside the metric and inference setup used across benchmarks.

A.3. Evaluation Details

  • Results use avg@k, the mean per-problem fraction of correct samples scaled to [0, 100], with benchmark averages equally weighted and reported over three seeds.At k = 1, avg@k is single-sample accuracy; evaluation uses vLLM with bf16 inference on NVIDIA A100 80GB GPUs.Source: §A.3 · p. 21
  • Math answers are extracted from the final boxed expression, with an “Answer:” fallback, then checked by string normalization and symbolic equivalence rather than a model judge.Qwen3 uses non-thinking mode; DeepSeek-R1-Distill students and their teachers use their default thinking templates.Source: §A.3 · p. 21
  • Code responses count as correct only when the extracted code passes every test case within the time limit.LiveCodeBench uses 342 v5–v6 problems, APPS uses specified introductory, competition, and interview test sets, and MBPP+ uses 378 problems with greedy decoding.Source: §A.3 · p. 21

A.4. Discussion

The discussion separates question generation, filtering, and downstream OPD as distinct pipeline stages. It also qualifies A2D’s reward as tailored to the downstream learning signal.

A.4. Discussion

  • A2D learns a question distribution for a target student–teacher pair, while filtering and the downstream OPD procedure act at separate pipeline stages.The synthetic question bank is prompt-only and can be used with alternative distillation procedures that accept prompt-only datasets.Source: §A.4 · pp. 21–22
  • The support-aware reward is designed for teacher supervision on the student’s candidate support, so other divergences, candidate supports, or supervision rules may require an adapted generation reward.The authors describe A2D as modular and leave exploration of alternative combinations to future work.Source: §A.4 · p. 22

A.5. Experimental Details and Additional Signal Validation

The appendix describes controlled analyses of question-bank signal and training outcomes, then states theoretical properties connecting normalized reward to compatible disagreement and the OPD update.

A.5. Experimental Details and Additional Signal Validation

  • The one-step learning-value measure is the decrease in full-vocabulary teacher-to-student KL on fixed probe responses after a single plain-SGD OPD step.The update uses four student responses and learning rate 3 × 10−5; its questions, prefixes, and KL direction differ from the probe evaluation.Source: §A.5 · p. 22
  • 2.21 points (95% CI [1.37, 3.06], p = 3.1 × 10−4) was the final-checkpoint high- versus low-divergence advantage in the original design; after balancing compatibility, it was 1.59 points (95% CI [0.63, 2.56], p = 0.005).Intermediate-checkpoint gains were 2.27 points (95% CI [0.62, 3.91], p = 0.013) in the original design and 2.79 points (95% CI [1.75, 3.82], p = 2.6 × 10−4) after balancing compatibility; all four intervals excluded zero.Source: Figure 10 · p. 22, §A.5 · p. 23
  • The appendix tests score-ranked banks against a same-sized random bank and uses high/low-divergence factorial designs, including a variant balanced on compatibility.Training uses a common 24.37M-token budget, while statistical inference for downstream comparisons treats each of three-seed training runs as the unit.Source: §A.5 · p. 23

B. Theoretical Properties of the Elicited-Signal Objective

  • High normalized reward certifies a nonzero fraction of response positions with both teacher–student discrepancy and teacher probability mass in the student candidate set.The theoretical properties characterize the elicited supervision and its connection to implemented OPD advantages without assuming synthesizer optimization converges.Source: B. Theoretical Properties of the… · p. 23

B.1. Compatible-disagreement coverage

The proposition links normalized reward to the fraction of response positions that jointly exhibit discrepancy and compatibility.

B.1. Compatible-disagreement coverage

  • A normalized reward above h certifies a nonzero fraction of positions with both discrepancy and compatibility.Normalization by L1−α is necessary because unnormalized reward also depends on response length.Source: §B.1 · p. 24
  • Expected coverage extends the pointwise guarantee by averaging over generated questions, student responses, response positions, and the batch-dependent diversity factor.The expectation follows from the pointwise inequality and nonnegativity.Source: §B.1 · p. 24

B.2. Compatibility and OPD advantages

The appendix relates compatibility to candidate-level OPD advantages and gives the question-generation and solving templates used in the pipeline.

B.2. Compatibility and OPD advantages

  • When scored with the same student and teacher distributions, s ≥ h implies log(ω/a) ≥ log h.Compatibility also adds a common offset to candidate log-ratios; its distributional interpretation relates teacher mass outside A to a total absolute difference of 1−ω.Source: §B.2 · p. 25
  • The analysis does not lower-bound parameter gradients or downstream improvement, and frozen-student coverage certificates need not persist as the student evolves during OPD.A fixed top-K set may also omit most student probability mass, which must be assessed separately through pS(A).Source: §B.2 · p. 25

C. Prompt Templates

  • Math and code synthesis use fixed instructions without persona conditioning or few-shot example problems.The templates specify the synthesizer and student-solving formats.Source: C. Prompt Templates · p. 27

SYNTHESIZER / MATH

  • The math synthesizer requests a novel, challenging, self-contained problem requiring several non-obvious reasoning steps, and outputs only the problem statement.It asks for a difficulty that fewer than 30% of advanced high-school students could solve.Source: SYNTHESIZER / MATH · p. 27

SYNTHESIZER / CODE

  • The code synthesizer requests a challenging, self-contained competitive-programming problem with input and output formats and input constraints, then outputs only its statement.Source: SYNTHESIZER / CODE · p. 27

MATH / Reward computation and discrepancy calibration

MATH / Downstream OPD on synthetic questions

MATH / Vanilla OPD on real questions

MATH / Evaluation suffix

The supplied evaluation suffix asks for boxed math answers, while the code-solving template specifies a Python-program output task.

MATH / Evaluation suffix

CODE / Student solving

  • The code-solving prompt asks for a correct Python program matching the specification and passing all tests, with no other output.Source: CODE / Student solving · p. 28
  • The code prompt directs programs to read stdin and write stdout, with the code enclosed in specified delimiters.Source: CODE / Student solving · p. 28

Related papers

Loading 2610.27450v1…